Evaluation of Sentence Selection for Speech Summarization

In the last several years, a number of papers have addressed the area of automatic speech summarization. Many of them have applied evaluation metrics adapted from those used in speech recognition research, rather than from those used in text summarization. We consider whether ASR-inspired evaluation metrics produce different results than those taken from text summarization, and why. We evaluate various standard summarizers as well as our own systems on a subset of the SWITCHBOARD spoken dialogue dataset with both kinds of metrics. We find a statistically significant departure between the two classes in their relative rank of these systems. Our preliminary conclusion is that considerably greater caution must be exercised when using ASR-based measures than we have witnessed to date in the speech summarization literature.